{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N7UbxgytUFhn"
      },
      "source": [
        "# Customized models and datasets for structured inputs\n",
        "\n",
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/HanXudong/fairlib/blob/main/tutorial/Structured_Inputs.ipynb)\n",
        "\n",
        "In this tutorial we will:\n",
        "- Show how to add a model for structural classification\n",
        "- Show how to add a dataloader with structured data preprocessing\n",
        "\n",
        "We will be using the Northpointe's Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) score, used in states like California and Florida.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Installation\n",
        "\n",
        "Again, the first step will be installing our libarary"
      ],
      "metadata": {
        "id": "qlKcs4pBV1Tc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install fairlib"
      ],
      "metadata": {
        "id": "FgkUn09buGwy",
        "outputId": "7c3ce2bc-d782-4c01-bd07-7a6d1d1d02dc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting fairlib\n",
            "  Downloading fairlib-0.0.9-py3-none-any.whl (85 kB)\n",
            "\u001b[K     |████████████████████████████████| 85 kB 2.6 MB/s \n",
            "\u001b[?25hRequirement already satisfied: PyYAML in /usr/local/lib/python3.7/dist-packages (from fairlib) (3.13)\n",
            "Requirement already satisfied: torch in /usr/local/lib/python3.7/dist-packages (from fairlib) (1.12.0+cu113)\n",
            "Collecting transformers\n",
            "  Downloading transformers-4.20.1-py3-none-any.whl (4.4 MB)\n",
            "\u001b[K     |████████████████████████████████| 4.4 MB 22.6 MB/s \n",
            "\u001b[?25hRequirement already satisfied: seaborn in /usr/local/lib/python3.7/dist-packages (from fairlib) (0.11.2)\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.7/dist-packages (from fairlib) (1.0.2)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from fairlib) (4.64.0)\n",
            "Requirement already satisfied: docopt in /usr/local/lib/python3.7/dist-packages (from fairlib) (0.6.2)\n",
            "Collecting sacremoses\n",
            "  Downloading sacremoses-0.0.53.tar.gz (880 kB)\n",
            "\u001b[K     |████████████████████████████████| 880 kB 62.2 MB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from fairlib) (1.21.6)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from fairlib) (1.3.5)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from fairlib) (3.2.2)\n",
            "Collecting pickle5\n",
            "  Downloading pickle5-0.0.12-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (256 kB)\n",
            "\u001b[K     |████████████████████████████████| 256 kB 47.9 MB/s \n",
            "\u001b[?25hRequirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->fairlib) (3.0.9)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->fairlib) (1.4.4)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->fairlib) (2.8.2)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->fairlib) (0.11.0)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from kiwisolver>=1.0.1->matplotlib->fairlib) (4.1.1)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.1->matplotlib->fairlib) (1.15.0)\n",
            "Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/dist-packages (from pandas->fairlib) (2022.1)\n",
            "Requirement already satisfied: regex in /usr/local/lib/python3.7/dist-packages (from sacremoses->fairlib) (2022.6.2)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->fairlib) (7.1.2)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->fairlib) (1.1.0)\n",
            "Requirement already satisfied: scipy>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->fairlib) (1.7.3)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->fairlib) (3.1.0)\n",
            "Collecting huggingface-hub<1.0,>=0.1.0\n",
            "  Downloading huggingface_hub-0.8.1-py3-none-any.whl (101 kB)\n",
            "\u001b[K     |████████████████████████████████| 101 kB 11.5 MB/s \n",
            "\u001b[?25hRequirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.7/dist-packages (from transformers->fairlib) (21.3)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from transformers->fairlib) (2.23.0)\n",
            "Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from transformers->fairlib) (4.12.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers->fairlib) (3.7.1)\n",
            "Collecting PyYAML\n",
            "  Downloading PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (596 kB)\n",
            "\u001b[K     |████████████████████████████████| 596 kB 11.8 MB/s \n",
            "\u001b[?25hCollecting tokenizers!=0.11.3,<0.13,>=0.11.1\n",
            "  Downloading tokenizers-0.12.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (6.6 MB)\n",
            "\u001b[K     |████████████████████████████████| 6.6 MB 33.9 MB/s \n",
            "\u001b[?25hRequirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->transformers->fairlib) (3.8.1)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->transformers->fairlib) (2.10)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->transformers->fairlib) (3.0.4)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->transformers->fairlib) (1.24.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->transformers->fairlib) (2022.6.15)\n",
            "Building wheels for collected packages: sacremoses\n",
            "  Building wheel for sacremoses (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for sacremoses: filename=sacremoses-0.0.53-py3-none-any.whl size=895260 sha256=9e7ef0aea29edc17a4bd4ba5d5f00a2eb51f47d5ed5f7ec133443d60dce94969\n",
            "  Stored in directory: /root/.cache/pip/wheels/87/39/dd/a83eeef36d0bf98e7a4d1933a4ad2d660295a40613079bafc9\n",
            "Successfully built sacremoses\n",
            "Installing collected packages: PyYAML, tokenizers, huggingface-hub, transformers, sacremoses, pickle5, fairlib\n",
            "  Attempting uninstall: PyYAML\n",
            "    Found existing installation: PyYAML 3.13\n",
            "    Uninstalling PyYAML-3.13:\n",
            "      Successfully uninstalled PyYAML-3.13\n",
            "Successfully installed PyYAML-6.0 fairlib-0.0.9 huggingface-hub-0.8.1 pickle5-0.0.12 sacremoses-0.0.53 tokenizers-0.12.1 transformers-4.20.1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "Xt6TeF2FVD0i"
      },
      "outputs": [],
      "source": [
        "import fairlib"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ePzbeMKO_eK1"
      },
      "source": [
        "## Download and preprocess the COMPAS dataset\n",
        "\n",
        "https://github.com/google-research/google-research/blob/master/group_agnostic_fairness/data_utils/CreateCompasDatasetFiles.ipynb"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from fairlib import datasets\n",
        "datasets.prepare_dataset(\"compas\", \"data\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 66,
          "referenced_widgets": [
            "f6137cf1210e40f28ed2ab98ecb2ba74",
            "8635022ff82d4fc48ac6ef965b29e043",
            "32be46c7c1574c17809338831434202f",
            "3e5331dd05a742729a83fb18f1d5e474",
            "a142bec177e6484eba9439ea554427ad",
            "ae6e5619acf84bdeb186166075f6a786",
            "f6be0b84185c4b2c9fdf15e9cb218f0f",
            "2dc9d1fd92674cde8844d53acadb6ef1",
            "a8113f07bea140c6a811176a20976707",
            "480f8a7ee5704f98ba463cf2395b63fe",
            "694dfb22b7704634a3aa34a8cf501444"
          ]
        },
        "id": "FBAw9U2MUKFu",
        "outputId": "87148261-5896-43b7-d9ab-b930fd2997d1"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "saving to /content/data/compas-scores-two-years.csv\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/1 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "f6137cf1210e40f28ed2ab98ecb2ba74"
            }
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "\n",
        "pd.read_pickle(\"data/COMPAS_dev.pkl\").keys()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tqMFGRh9lWK8",
        "outputId": "bef11c91-b28a-4a39-b734-b1e8a9491e9f"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Index(['juv_fel_count', 'juv_misd_count', 'juv_other_count', 'priors_count',\n",
              "       'age', 'c_charge_degree_0', 'c_charge_degree_1', 'c_charge_desc_262',\n",
              "       'c_charge_desc_43', 'c_charge_desc_55',\n",
              "       ...\n",
              "       'c_charge_desc_323', 'c_charge_desc_66', 'c_charge_desc_40',\n",
              "       'c_charge_desc_33', 'age_cat_1', 'age_cat_0', 'age_cat_2', 'sex',\n",
              "       'race', 'is_recid'],\n",
              "      dtype='object', length=450)"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Train Models"
      ],
      "metadata": {
        "id": "vZDfxgHaUa9m"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "3qOYUt8eUDT2"
      },
      "outputs": [],
      "source": [
        "from fairlib import networks, BaseOptions, dataloaders\n",
        "import torch"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "WIQgZgVgUDT2"
      },
      "outputs": [],
      "source": [
        "Shared_options = {\n",
        "    # The name of the dataset, correponding dataloader will be used,\n",
        "    \"dataset\":  \"COMPAS\",\n",
        "\n",
        "    # Specifiy the path to the input data\n",
        "    \"data_dir\": \"./data\",\n",
        "\n",
        "    # Device for computing, -1 is the cpu\n",
        "    \"device_id\": -1,\n",
        "\n",
        "    # The default path for saving experimental results\n",
        "    \"results_dir\":  r\"results\",\n",
        "\n",
        "    # The same as the dataset\n",
        "    \"project_dir\":  r\"dev\",\n",
        "\n",
        "    # We will focusing on TPR GAP, implying the Equalized Odds for binay classification.\n",
        "    \"GAP_metric_name\":  \"TPR_GAP\",\n",
        "\n",
        "    # The overall performance will be measured as accuracy\n",
        "    \"Performance_metric_name\":  \"accuracy\",\n",
        "\n",
        "    # Model selections are based on DTO\n",
        "    \"selection_criterion\":  \"DTO\",\n",
        "\n",
        "    # Default dirs for saving checkpoints\n",
        "    \"checkpoint_dir\":   \"models\",\n",
        "    \"checkpoint_name\":  \"checkpoint_epoch\",\n",
        "\n",
        "\n",
        "    \"n_jobs\":   1,\n",
        "}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-fAarStJUDT2",
        "outputId": "bb1d5691-a784-4a68-dfee-1099853ea85b"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:root:Unexpected args: ['-f', '/root/.local/share/jupyter/runtime/kernel-1cc80252-6b1d-427f-a63d-8b70911dfc3b.json']\n",
            "INFO:root:Logging to ./results/dev/COMPAS/vanilla/output.log\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2022-07-21 07:10:35 [INFO ]  ======================================== 2022-07-21 07:10:35 ========================================\n",
            "2022-07-21 07:10:35 [INFO ]  Base directory is ./results/dev/COMPAS/vanilla\n",
            "2022-07-21 07:10:35 [INFO ]  Exception type : AssertionError \n",
            "2022-07-21 07:10:35 [INFO ]  Exception message : Not implemented\n",
            "2022-07-21 07:10:35 [INFO ]  Stack trace : ['File : /usr/local/lib/python3.7/dist-packages/fairlib/src/base_options.py , Line : 486, Func.Name : set_state, Message : train_iterator, dev_iterator, test_iterator = dataloaders.get_dataloaders(state)', 'File : /usr/local/lib/python3.7/dist-packages/fairlib/src/dataloaders/__init__.py , Line : 40, Func.Name : get_dataloaders, Message : ], \"Not implemented\"']\n",
            "2022-07-21 07:10:35 [INFO ]  dataloaders need to be initialized!\n"
          ]
        }
      ],
      "source": [
        "args = {\n",
        "    \"dataset\":Shared_options[\"dataset\"], \n",
        "    \"data_dir\":Shared_options[\"data_dir\"],\n",
        "    \"device_id\":Shared_options[\"device_id\"],\n",
        "\n",
        "    # Give a name to the exp, which will be used in the path\n",
        "    \"exp_id\":\"vanilla\",\n",
        "\n",
        "    \"emb_size\": 450-3,\n",
        "    \"lr\": 0.001,\n",
        "    \"batch_size\": 128,\n",
        "    \"hidden_size\": 32,\n",
        "    \"n_hidden\": 1,\n",
        "    \"activation_function\": \"ReLu\",\n",
        "\n",
        "    \"num_classes\": 2,\n",
        "    \"num_groups\": 3, # Balck; White; and Other\n",
        "}\n",
        "\n",
        "# Init the argument\n",
        "options = BaseOptions()\n",
        "state = options.get_state(args=args, silence=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Customize dataset loader"
      ],
      "metadata": {
        "id": "10y7s7MgV6CB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "import pandas as pd\n",
        "import numpy as np"
      ],
      "metadata": {
        "id": "gEWRx22GWAQG"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "1L9WAC7jUDT3"
      },
      "outputs": [],
      "source": [
        "class CustomizedDataset(dataloaders.utils.BaseDataset):\n",
        "\n",
        "    def load_data(self):\n",
        "\n",
        "        self.data_dir = os.path.join(self.args.data_dir, \"COMPAS_{}.pkl\".format(self.split))\n",
        "\n",
        "        data = pd.read_pickle(self.data_dir)\n",
        "\n",
        "        self.X = data.drop(['sex', 'race', 'is_recid'], axis=1).to_numpy().astype(np.float32)\n",
        "        self.y = list(data[\"is_recid\"])\n",
        "        self.protected_label = list(data[\"race\"])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xXc659mpUDT3",
        "outputId": "63218915-792a-4a1f-83fb-f829daced7f2"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Loaded data shapes: (4544, 447), (4544,), (4544,)\n",
            "Loaded data shapes: (505, 447), (505,), (505,)\n",
            "Loaded data shapes: (2165, 447), (2165,), (2165,)\n"
          ]
        }
      ],
      "source": [
        "customized_train_data = CustomizedDataset(args=state, split=\"train\")\n",
        "customized_dev_data = CustomizedDataset(args=state, split=\"dev\")\n",
        "customized_test_data = CustomizedDataset(args=state, split=\"test\")\n",
        "\n",
        "# DataLoader Parameters\n",
        "tran_dataloader_params = {\n",
        "        'batch_size': state.batch_size,\n",
        "        'shuffle': True,\n",
        "        'num_workers': state.num_workers}\n",
        "\n",
        "eval_dataloader_params = {\n",
        "        'batch_size': state.test_batch_size,\n",
        "        'shuffle': False,\n",
        "        'num_workers': state.num_workers}\n",
        "\n",
        "# init dataloader\n",
        "customized_training_generator = torch.utils.data.DataLoader(customized_train_data, **tran_dataloader_params)\n",
        "customized_validation_generator = torch.utils.data.DataLoader(customized_dev_data, **eval_dataloader_params)\n",
        "customized_test_generator = torch.utils.data.DataLoader(customized_test_data, **eval_dataloader_params)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SGfihIUqUDT3",
        "outputId": "14afc39b-cb19-41bb-ee45-c9888b8e1d6c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2022-07-21 07:10:35 [INFO ]  MLP( \n",
            "2022-07-21 07:10:35 [INFO ]    (output_layer): Linear(in_features=32, out_features=2, bias=True)\n",
            "2022-07-21 07:10:35 [INFO ]    (AF): ReLU()\n",
            "2022-07-21 07:10:35 [INFO ]    (hidden_layers): ModuleList(\n",
            "2022-07-21 07:10:35 [INFO ]      (0): Linear(in_features=447, out_features=32, bias=True)\n",
            "2022-07-21 07:10:35 [INFO ]      (1): ReLU()\n",
            "2022-07-21 07:10:35 [INFO ]    )\n",
            "2022-07-21 07:10:35 [INFO ]    (criterion): CrossEntropyLoss()\n",
            "2022-07-21 07:10:35 [INFO ]  )\n",
            "2022-07-21 07:10:35 [INFO ]  Total number of parameters: 14402 \n",
            "\n"
          ]
        }
      ],
      "source": [
        "model = networks.classifier.MLP(state)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BzHN7paxUDT4",
        "outputId": "239317f7-c471-4605-fb01-5f61e1d89e43"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2022-07-21 07:10:35 [INFO ]  Epoch:    0 [      0/   4544 ( 0%)]\tLoss: 0.6946\t Data Time: 0.05s\tTrain Time: 0.12s\n",
            "2022-07-21 07:10:35 [INFO ]  Evaluation at Epoch 0\n",
            "2022-07-21 07:10:35 [INFO ]  Validation accuracy: 65.35\tmacro_fscore: 64.41\tmicro_fscore: 65.35\tTPR_GAP: 37.09\tFPR_GAP: 37.09\tPPR_GAP: 38.61\t\n",
            "2022-07-21 07:10:35 [INFO ]  Test accuracy: 67.30\tmacro_fscore: 66.29\tmicro_fscore: 67.30\tTPR_GAP: 28.96\tFPR_GAP: 28.96\tPPR_GAP: 33.07\t\n",
            "2022-07-21 07:10:35 [INFO ]  Epoch:    1 [      0/   4544 ( 0%)]\tLoss: 0.6375\t Data Time: 0.00s\tTrain Time: 0.00s\n",
            "2022-07-21 07:10:35 [INFO ]  Evaluation at Epoch 1\n",
            "2022-07-21 07:10:35 [INFO ]  Validation accuracy: 66.14\tmacro_fscore: 65.92\tmicro_fscore: 66.14\tTPR_GAP: 30.20\tFPR_GAP: 30.20\tPPR_GAP: 32.54\t\n",
            "2022-07-21 07:10:35 [INFO ]  Test accuracy: 67.53\tmacro_fscore: 67.12\tmicro_fscore: 67.53\tTPR_GAP: 31.45\tFPR_GAP: 31.45\tPPR_GAP: 36.71\t\n",
            "2022-07-21 07:10:35 [INFO ]  Epoch:    2 [      0/   4544 ( 0%)]\tLoss: 0.5948\t Data Time: 0.01s\tTrain Time: 0.00s\n",
            "2022-07-21 07:10:36 [INFO ]  Evaluation at Epoch 2\n",
            "2022-07-21 07:10:36 [INFO ]  Validation accuracy: 66.14\tmacro_fscore: 66.00\tmicro_fscore: 66.14\tTPR_GAP: 31.74\tFPR_GAP: 31.74\tPPR_GAP: 34.83\t\n",
            "2022-07-21 07:10:36 [INFO ]  Test accuracy: 68.55\tmacro_fscore: 68.24\tmicro_fscore: 68.55\tTPR_GAP: 32.02\tFPR_GAP: 32.02\tPPR_GAP: 38.27\t\n",
            "2022-07-21 07:10:36 [INFO ]  Epoch:    3 [      0/   4544 ( 0%)]\tLoss: 0.6298\t Data Time: 0.01s\tTrain Time: 0.00s\n",
            "2022-07-21 07:10:36 [INFO ]  Epochs since last improvement: 1\n",
            "2022-07-21 07:10:36 [INFO ]  Evaluation at Epoch 3\n",
            "2022-07-21 07:10:36 [INFO ]  Validation accuracy: 65.94\tmacro_fscore: 65.71\tmicro_fscore: 65.94\tTPR_GAP: 35.41\tFPR_GAP: 35.41\tPPR_GAP: 38.06\t\n",
            "2022-07-21 07:10:36 [INFO ]  Test accuracy: 68.96\tmacro_fscore: 68.54\tmicro_fscore: 68.96\tTPR_GAP: 32.02\tFPR_GAP: 32.02\tPPR_GAP: 37.86\t\n",
            "2022-07-21 07:10:36 [INFO ]  Epoch:    4 [      0/   4544 ( 0%)]\tLoss: 0.5868\t Data Time: 0.00s\tTrain Time: 0.00s\n",
            "2022-07-21 07:10:36 [INFO ]  Epochs since last improvement: 2\n",
            "2022-07-21 07:10:36 [INFO ]  Evaluation at Epoch 4\n",
            "2022-07-21 07:10:36 [INFO ]  Validation accuracy: 65.35\tmacro_fscore: 65.32\tmicro_fscore: 65.35\tTPR_GAP: 35.10\tFPR_GAP: 35.10\tPPR_GAP: 38.32\t\n",
            "2022-07-21 07:10:36 [INFO ]  Test accuracy: 68.91\tmacro_fscore: 68.80\tmicro_fscore: 68.91\tTPR_GAP: 33.54\tFPR_GAP: 33.54\tPPR_GAP: 40.45\t\n",
            "2022-07-21 07:10:36 [INFO ]  Epoch:    5 [      0/   4544 ( 0%)]\tLoss: 0.5572\t Data Time: 0.00s\tTrain Time: 0.00s\n",
            "2022-07-21 07:10:36 [INFO ]  Epochs since last improvement: 3\n",
            "2022-07-21 07:10:36 [INFO ]  Evaluation at Epoch 5\n",
            "2022-07-21 07:10:36 [INFO ]  Validation accuracy: 65.54\tmacro_fscore: 65.24\tmicro_fscore: 65.54\tTPR_GAP: 35.02\tFPR_GAP: 35.02\tPPR_GAP: 37.96\t\n",
            "2022-07-21 07:10:36 [INFO ]  Test accuracy: 68.96\tmacro_fscore: 68.47\tmicro_fscore: 68.96\tTPR_GAP: 30.91\tFPR_GAP: 30.91\tPPR_GAP: 36.59\t\n",
            "2022-07-21 07:10:36 [INFO ]  Epoch:    6 [      0/   4544 ( 0%)]\tLoss: 0.5651\t Data Time: 0.00s\tTrain Time: 0.00s\n",
            "2022-07-21 07:10:36 [INFO ]  Epochs since last improvement: 4\n",
            "2022-07-21 07:10:36 [INFO ]  Evaluation at Epoch 6\n",
            "2022-07-21 07:10:36 [INFO ]  Validation accuracy: 64.36\tmacro_fscore: 64.22\tmicro_fscore: 64.36\tTPR_GAP: 34.11\tFPR_GAP: 34.11\tPPR_GAP: 36.69\t\n",
            "2022-07-21 07:10:36 [INFO ]  Test accuracy: 69.01\tmacro_fscore: 68.75\tmicro_fscore: 69.01\tTPR_GAP: 30.58\tFPR_GAP: 30.58\tPPR_GAP: 37.35\t\n",
            "2022-07-21 07:10:36 [INFO ]  Epoch:    7 [      0/   4544 ( 0%)]\tLoss: 0.6300\t Data Time: 0.00s\tTrain Time: 0.00s\n",
            "2022-07-21 07:10:36 [INFO ]  Epochs since last improvement: 5\n",
            "2022-07-21 07:10:36 [INFO ]  Evaluation at Epoch 7\n",
            "2022-07-21 07:10:37 [INFO ]  Validation accuracy: 64.75\tmacro_fscore: 64.75\tmicro_fscore: 64.75\tTPR_GAP: 31.62\tFPR_GAP: 31.62\tPPR_GAP: 34.74\t\n",
            "2022-07-21 07:10:37 [INFO ]  Test accuracy: 68.68\tmacro_fscore: 68.59\tmicro_fscore: 68.68\tTPR_GAP: 31.42\tFPR_GAP: 31.42\tPPR_GAP: 38.69\t\n"
          ]
        }
      ],
      "source": [
        "model.train_self(\n",
        "    train_generator = customized_training_generator,\n",
        "    dev_generator = customized_validation_generator,\n",
        "    test_generator = customized_test_generator,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xmqdlUG6UDT4",
        "outputId": "f1ff232b-fb2f-4769-eee7-21ce8302274c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2022-07-21 07:10:37 [INFO ]  Unexpected args: ['-f', '/root/.local/share/jupyter/runtime/kernel-1cc80252-6b1d-427f-a63d-8b70911dfc3b.json']\n",
            "2022-07-21 07:10:37 [INFO ]  Logging to ./results/dev/COMPAS/BT_Adv/output.log\n",
            "2022-07-21 07:10:37 [INFO ]  ======================================== 2022-07-21 07:10:37 ========================================\n",
            "2022-07-21 07:10:37 [INFO ]  Base directory is ./results/dev/COMPAS/BT_Adv\n",
            "2022-07-21 07:10:37 [INFO ]  Exception type : AssertionError \n",
            "2022-07-21 07:10:37 [INFO ]  Exception message : Not implemented\n",
            "2022-07-21 07:10:37 [INFO ]  Stack trace : ['File : /usr/local/lib/python3.7/dist-packages/fairlib/src/base_options.py , Line : 486, Func.Name : set_state, Message : train_iterator, dev_iterator, test_iterator = dataloaders.get_dataloaders(state)', 'File : /usr/local/lib/python3.7/dist-packages/fairlib/src/dataloaders/__init__.py , Line : 40, Func.Name : get_dataloaders, Message : ], \"Not implemented\"']\n",
            "2022-07-21 07:10:37 [INFO ]  dataloaders need to be initialized!\n",
            "2022-07-21 07:10:37 [INFO ]  SubDiscriminator( \n",
            "2022-07-21 07:10:37 [INFO ]    (grad_rev): GradientReversal()\n",
            "2022-07-21 07:10:37 [INFO ]    (output_layer): Linear(in_features=300, out_features=3, bias=True)\n",
            "2022-07-21 07:10:37 [INFO ]    (AF): ReLU()\n",
            "2022-07-21 07:10:37 [INFO ]    (hidden_layers): ModuleList(\n",
            "2022-07-21 07:10:37 [INFO ]      (0): Linear(in_features=32, out_features=300, bias=True)\n",
            "2022-07-21 07:10:37 [INFO ]      (1): ReLU()\n",
            "2022-07-21 07:10:37 [INFO ]      (2): Linear(in_features=300, out_features=300, bias=True)\n",
            "2022-07-21 07:10:37 [INFO ]      (3): ReLU()\n",
            "2022-07-21 07:10:37 [INFO ]    )\n",
            "2022-07-21 07:10:37 [INFO ]    (criterion): CrossEntropyLoss()\n",
            "2022-07-21 07:10:37 [INFO ]  )\n",
            "2022-07-21 07:10:37 [INFO ]  Total number of parameters: 101103 \n",
            "\n",
            "2022-07-21 07:10:37 [INFO ]  Discriminator built!\n",
            "Loaded data shapes: (4542, 447), (4542,), (4542,)\n",
            "Loaded data shapes: (505, 447), (505,), (505,)\n",
            "Loaded data shapes: (2165, 447), (2165,), (2165,)\n",
            "2022-07-21 07:10:37 [INFO ]  MLP( \n",
            "2022-07-21 07:10:37 [INFO ]    (output_layer): Linear(in_features=32, out_features=2, bias=True)\n",
            "2022-07-21 07:10:37 [INFO ]    (AF): ReLU()\n",
            "2022-07-21 07:10:37 [INFO ]    (hidden_layers): ModuleList(\n",
            "2022-07-21 07:10:37 [INFO ]      (0): Linear(in_features=447, out_features=32, bias=True)\n",
            "2022-07-21 07:10:37 [INFO ]      (1): ReLU()\n",
            "2022-07-21 07:10:37 [INFO ]    )\n",
            "2022-07-21 07:10:37 [INFO ]    (criterion): CrossEntropyLoss()\n",
            "2022-07-21 07:10:37 [INFO ]  )\n",
            "2022-07-21 07:10:37 [INFO ]  Total number of parameters: 14402 \n",
            "\n"
          ]
        }
      ],
      "source": [
        "debiasing_args = {\n",
        "    \"dataset\":Shared_options[\"dataset\"], \n",
        "    \"data_dir\":Shared_options[\"data_dir\"],\n",
        "    \"device_id\":Shared_options[\"device_id\"],\n",
        "\n",
        "    # Give a name to the exp, which will be used in the path\n",
        "    \"exp_id\":\"BT_Adv\",\n",
        "\n",
        "    \"emb_size\": 450-3,\n",
        "    \"lr\": 0.001,\n",
        "    \"batch_size\": 128,\n",
        "    \"hidden_size\": 32,\n",
        "    \"n_hidden\": 1,\n",
        "    \"activation_function\": \"ReLu\",\n",
        "\n",
        "    \"num_classes\": 2,\n",
        "    \"num_groups\": 3, # Balck; White; and Other\n",
        "\n",
        "    # Perform adversarial training if True\n",
        "    \"adv_debiasing\":True,\n",
        "\n",
        "    # Specify the hyperparameters for Balanced Training\n",
        "    \"BT\":\"Resampling\",\n",
        "    \"BTObj\":\"EO\",\n",
        "}\n",
        "\n",
        "# Init the argument\n",
        "debias_options = BaseOptions()\n",
        "debias_state = debias_options.get_state(args=debiasing_args, silence=True)\n",
        "\n",
        "customized_train_data = CustomizedDataset(args=debias_state, split=\"train\")\n",
        "customized_dev_data = CustomizedDataset(args=debias_state, split=\"dev\")\n",
        "customized_test_data = CustomizedDataset(args=debias_state, split=\"test\")\n",
        "\n",
        "# DataLoader Parameters\n",
        "tran_dataloader_params = {\n",
        "        'batch_size': state.batch_size,\n",
        "        'shuffle': True,\n",
        "        'num_workers': state.num_workers}\n",
        "\n",
        "eval_dataloader_params = {\n",
        "        'batch_size': state.test_batch_size,\n",
        "        'shuffle': False,\n",
        "        'num_workers': state.num_workers}\n",
        "\n",
        "# init dataloader\n",
        "customized_training_generator = torch.utils.data.DataLoader(customized_train_data, **tran_dataloader_params)\n",
        "customized_validation_generator = torch.utils.data.DataLoader(customized_dev_data, **eval_dataloader_params)\n",
        "customized_test_generator = torch.utils.data.DataLoader(customized_test_data, **eval_dataloader_params)\n",
        "\n",
        "debias_model = networks.classifier.MLP(debias_state)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jLJQQzilUDT4",
        "outputId": "e4780e65-0666-49f4-ea28-6c48e97a0807"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2022-07-21 07:10:37 [INFO ]  Epoch:    0 [      0/   4542 ( 0%)]\tLoss: -0.4072\t Data Time: 0.00s\tTrain Time: 0.05s\n",
            "2022-07-21 07:10:37 [INFO ]  Evaluation at Epoch 0\n",
            "2022-07-21 07:10:38 [INFO ]  Validation accuracy: 65.54\tmacro_fscore: 65.27\tmicro_fscore: 65.54\tTPR_GAP: 32.32\tFPR_GAP: 32.32\tPPR_GAP: 35.63\t\n",
            "2022-07-21 07:10:38 [INFO ]  Test accuracy: 67.44\tmacro_fscore: 67.00\tmicro_fscore: 67.44\tTPR_GAP: 30.82\tFPR_GAP: 30.82\tPPR_GAP: 36.06\t\n",
            "2022-07-21 07:10:38 [INFO ]  Epoch:    1 [      0/   4542 ( 0%)]\tLoss: -0.4403\t Data Time: 0.00s\tTrain Time: 0.02s\n",
            "2022-07-21 07:10:38 [INFO ]  Evaluation at Epoch 1\n",
            "2022-07-21 07:10:38 [INFO ]  Validation accuracy: 64.36\tmacro_fscore: 64.04\tmicro_fscore: 64.36\tTPR_GAP: 31.50\tFPR_GAP: 31.50\tPPR_GAP: 34.76\t\n",
            "2022-07-21 07:10:38 [INFO ]  Test accuracy: 67.67\tmacro_fscore: 67.06\tmicro_fscore: 67.67\tTPR_GAP: 28.13\tFPR_GAP: 28.13\tPPR_GAP: 34.07\t\n",
            "2022-07-21 07:10:38 [INFO ]  Epoch:    2 [      0/   4542 ( 0%)]\tLoss: -0.4799\t Data Time: 0.00s\tTrain Time: 0.03s\n",
            "2022-07-21 07:10:39 [INFO ]  Evaluation at Epoch 2\n",
            "2022-07-21 07:10:39 [INFO ]  Validation accuracy: 61.58\tmacro_fscore: 61.53\tmicro_fscore: 61.58\tTPR_GAP: 32.47\tFPR_GAP: 32.47\tPPR_GAP: 34.93\t\n",
            "2022-07-21 07:10:39 [INFO ]  Test accuracy: 66.65\tmacro_fscore: 66.41\tmicro_fscore: 66.65\tTPR_GAP: 27.35\tFPR_GAP: 27.36\tPPR_GAP: 33.06\t\n",
            "2022-07-21 07:10:39 [INFO ]  Epoch:    3 [      0/   4542 ( 0%)]\tLoss: -0.4201\t Data Time: 0.00s\tTrain Time: 0.02s\n",
            "2022-07-21 07:10:40 [INFO ]  Evaluation at Epoch 3\n",
            "2022-07-21 07:10:40 [INFO ]  Validation accuracy: 62.97\tmacro_fscore: 62.76\tmicro_fscore: 62.97\tTPR_GAP: 35.73\tFPR_GAP: 35.73\tPPR_GAP: 38.56\t\n",
            "2022-07-21 07:10:40 [INFO ]  Test accuracy: 67.62\tmacro_fscore: 67.05\tmicro_fscore: 67.62\tTPR_GAP: 26.93\tFPR_GAP: 26.93\tPPR_GAP: 33.23\t\n",
            "2022-07-21 07:10:40 [INFO ]  Epoch:    4 [      0/   4542 ( 0%)]\tLoss: -0.5529\t Data Time: 0.00s\tTrain Time: 0.01s\n",
            "2022-07-21 07:10:41 [INFO ]  Epochs since last improvement: 1\n",
            "2022-07-21 07:10:41 [INFO ]  Evaluation at Epoch 4\n",
            "2022-07-21 07:10:41 [INFO ]  Validation accuracy: 61.98\tmacro_fscore: 61.78\tmicro_fscore: 61.98\tTPR_GAP: 35.31\tFPR_GAP: 35.31\tPPR_GAP: 37.22\t\n",
            "2022-07-21 07:10:41 [INFO ]  Test accuracy: 67.67\tmacro_fscore: 67.16\tmicro_fscore: 67.67\tTPR_GAP: 25.47\tFPR_GAP: 25.47\tPPR_GAP: 31.31\t\n",
            "2022-07-21 07:10:41 [INFO ]  Epoch:    5 [      0/   4542 ( 0%)]\tLoss: -0.4623\t Data Time: 0.00s\tTrain Time: 0.02s\n",
            "2022-07-21 07:10:42 [INFO ]  Evaluation at Epoch 5\n",
            "2022-07-21 07:10:42 [INFO ]  Validation accuracy: 63.37\tmacro_fscore: 63.24\tmicro_fscore: 63.37\tTPR_GAP: 32.46\tFPR_GAP: 32.46\tPPR_GAP: 34.72\t\n",
            "2022-07-21 07:10:42 [INFO ]  Test accuracy: 67.02\tmacro_fscore: 66.56\tmicro_fscore: 67.02\tTPR_GAP: 26.19\tFPR_GAP: 26.19\tPPR_GAP: 32.26\t\n",
            "2022-07-21 07:10:42 [INFO ]  Epoch:    6 [      0/   4542 ( 0%)]\tLoss: -0.4727\t Data Time: 0.00s\tTrain Time: 0.01s\n",
            "2022-07-21 07:10:43 [INFO ]  Epochs since last improvement: 1\n",
            "2022-07-21 07:10:43 [INFO ]  Evaluation at Epoch 6\n",
            "2022-07-21 07:10:43 [INFO ]  Validation accuracy: 65.15\tmacro_fscore: 65.05\tmicro_fscore: 65.15\tTPR_GAP: 32.60\tFPR_GAP: 32.60\tPPR_GAP: 35.29\t\n",
            "2022-07-21 07:10:43 [INFO ]  Test accuracy: 66.70\tmacro_fscore: 66.36\tmicro_fscore: 66.70\tTPR_GAP: 27.40\tFPR_GAP: 27.40\tPPR_GAP: 32.54\t\n",
            "2022-07-21 07:10:43 [INFO ]  Epoch:    7 [      0/   4542 ( 0%)]\tLoss: -0.4569\t Data Time: 0.00s\tTrain Time: 0.02s\n",
            "2022-07-21 07:10:44 [INFO ]  Epochs since last improvement: 2\n",
            "2022-07-21 07:10:44 [INFO ]  Evaluation at Epoch 7\n",
            "2022-07-21 07:10:44 [INFO ]  Validation accuracy: 65.74\tmacro_fscore: 65.69\tmicro_fscore: 65.74\tTPR_GAP: 32.00\tFPR_GAP: 32.00\tPPR_GAP: 35.02\t\n",
            "2022-07-21 07:10:44 [INFO ]  Test accuracy: 66.84\tmacro_fscore: 66.62\tmicro_fscore: 66.84\tTPR_GAP: 27.97\tFPR_GAP: 27.97\tPPR_GAP: 33.59\t\n",
            "2022-07-21 07:10:44 [INFO ]  Epoch:    8 [      0/   4542 ( 0%)]\tLoss: -0.4861\t Data Time: 0.00s\tTrain Time: 0.01s\n",
            "2022-07-21 07:10:44 [INFO ]  Epochs since last improvement: 3\n",
            "2022-07-21 07:10:44 [INFO ]  Evaluation at Epoch 8\n",
            "2022-07-21 07:10:44 [INFO ]  Validation accuracy: 64.36\tmacro_fscore: 64.16\tmicro_fscore: 64.36\tTPR_GAP: 39.90\tFPR_GAP: 39.90\tPPR_GAP: 42.72\t\n",
            "2022-07-21 07:10:44 [INFO ]  Test accuracy: 66.28\tmacro_fscore: 65.64\tmicro_fscore: 66.28\tTPR_GAP: 24.97\tFPR_GAP: 24.97\tPPR_GAP: 29.19\t\n",
            "2022-07-21 07:10:44 [INFO ]  Epoch:    9 [      0/   4542 ( 0%)]\tLoss: -0.5028\t Data Time: 0.00s\tTrain Time: 0.01s\n",
            "2022-07-21 07:10:45 [INFO ]  Epochs since last improvement: 4\n",
            "2022-07-21 07:10:45 [INFO ]  Evaluation at Epoch 9\n",
            "2022-07-21 07:10:45 [INFO ]  Validation accuracy: 63.76\tmacro_fscore: 63.67\tmicro_fscore: 63.76\tTPR_GAP: 31.83\tFPR_GAP: 31.83\tPPR_GAP: 34.30\t\n",
            "2022-07-21 07:10:45 [INFO ]  Test accuracy: 66.61\tmacro_fscore: 66.15\tmicro_fscore: 66.61\tTPR_GAP: 25.55\tFPR_GAP: 25.55\tPPR_GAP: 29.76\t\n",
            "2022-07-21 07:10:45 [INFO ]  Epoch:   10 [      0/   4542 ( 0%)]\tLoss: -0.5265\t Data Time: 0.00s\tTrain Time: 0.01s\n",
            "2022-07-21 07:10:46 [INFO ]  Epochs since last improvement: 5\n",
            "2022-07-21 07:10:46 [INFO ]  Evaluation at Epoch 10\n",
            "2022-07-21 07:10:46 [INFO ]  Validation accuracy: 64.55\tmacro_fscore: 64.55\tmicro_fscore: 64.55\tTPR_GAP: 30.83\tFPR_GAP: 30.83\tPPR_GAP: 32.99\t\n",
            "2022-07-21 07:10:46 [INFO ]  Test accuracy: 66.28\tmacro_fscore: 66.11\tmicro_fscore: 66.28\tTPR_GAP: 26.50\tFPR_GAP: 26.50\tPPR_GAP: 32.48\t\n"
          ]
        }
      ],
      "source": [
        "debias_model.train_self(\n",
        "    train_generator = customized_training_generator,\n",
        "    dev_generator = customized_validation_generator,\n",
        "    test_generator = customized_test_generator,\n",
        ")"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "Structured_Inputs.ipynb",
      "provenance": [],
      "toc_visible": true
    },
    "interpreter": {
      "hash": "086b6e2ac5931a4f7a51b812f0475290a145e7d51f3e22b53c52adf5c273fe30"
    },
    "kernelspec": {
      "display_name": "Python 3.7.12 ('py37')",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.12"
    },
    "orig_nbformat": 4,
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "f6137cf1210e40f28ed2ab98ecb2ba74": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_8635022ff82d4fc48ac6ef965b29e043",
              "IPY_MODEL_32be46c7c1574c17809338831434202f",
              "IPY_MODEL_3e5331dd05a742729a83fb18f1d5e474"
            ],
            "layout": "IPY_MODEL_a142bec177e6484eba9439ea554427ad"
          }
        },
        "8635022ff82d4fc48ac6ef965b29e043": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ae6e5619acf84bdeb186166075f6a786",
            "placeholder": "​",
            "style": "IPY_MODEL_f6be0b84185c4b2c9fdf15e9cb218f0f",
            "value": "100%"
          }
        },
        "32be46c7c1574c17809338831434202f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2dc9d1fd92674cde8844d53acadb6ef1",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_a8113f07bea140c6a811176a20976707",
            "value": 1
          }
        },
        "3e5331dd05a742729a83fb18f1d5e474": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_480f8a7ee5704f98ba463cf2395b63fe",
            "placeholder": "​",
            "style": "IPY_MODEL_694dfb22b7704634a3aa34a8cf501444",
            "value": " 1/1 [00:00&lt;00:00,  6.07it/s]"
          }
        },
        "a142bec177e6484eba9439ea554427ad": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ae6e5619acf84bdeb186166075f6a786": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f6be0b84185c4b2c9fdf15e9cb218f0f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "2dc9d1fd92674cde8844d53acadb6ef1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a8113f07bea140c6a811176a20976707": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "480f8a7ee5704f98ba463cf2395b63fe": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "694dfb22b7704634a3aa34a8cf501444": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}